Strong correlations at topological insulator surfaces and the breakdown of the 

bulk-boundary correspondence 
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The criteria for strong correlations on surfaces of three-dimensional topological insulators are dis- 
cussed. Usually, the Coulomb repulsion at such surfaces is too weak for driving a phase transition 
to a strongly correlated regime. I discuss a mechanism and possibilities of its experimental imple- 
mentation by which the strength of the Coulomb interaction can be tuned over a wide range. In the 
strongly interacting regime, the surface states are gapped, even though the topological classification 
of the bulk band structure predicts gapless surface states. 
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The field of topological insulators (TI) started a few 
years ago with the investigation of spin-orbit interactions 
(SOI) ingraphene where it was shown that graphene 
with an artificially large SOI is insulating in the bulk, 
but has perfectly conducting channels at its boundaries. 
It was quickly noticed 0, 0] that the presence of these 
boundary states is intimately connected to the topolog- 
ical properties of the bulk material (bulk-boundary cor- 
respondence); a topological classification of the insulat- 
ing band structure - a bulk property - gives rise to two 
distinct material classes: trivial and nontrivial insula- 
tors. The latter must have perfectly conducting bound- 
ary channels which are robust against all time-reversal 
invariant single-particle perturbations In this early 
stage of topological classifications the electron-electron 
interaction played essentially no role and the theory was 
based on band structure arguments. 

More recently, however, researchers noticed that the 
topological classification of insulators may be extended to 
include the Coulomb repulsion [Hj . The possibility of per- 
forming unbiased and numerically exact quantum Monte- 
Carlo simulations of the Kane-Mclc-Hubbard model @ 
- a paradigmatic model for a strongly interacting two- 
dimensional (2D) TI - ignited much research exploring 
the interplay of interactions and topology. The topologi- 
cal bulk properties can be destroyed by a strong Coulomb 
repulsion, which may drive a quantum phase transition 
to a time-reversal symmetry-broken state 0. But also 
the direct effect of interactions on the helical edge states 
has been studied It turned out that, while the heli- 
cal edge states are robust against weak interactions [1], 
strong interactions lead to dramatic changes of the single- 
particle properties near the edges 0]. 

The investigation of correlations in TIs were largely re- 
stricted to 2D TIs with ID states at their edges. Strong 
correlations at surfaces of 3D TIs have not yet received 
much attention. This might be due to the rather large 
interaction strengths that are usually needed for driving 
a phase transition of those surfaces. In this Letter, I 
show how the surface states can be tuned to a strongly 
interacting regime, while the bulk stays practically non- 



interacting. The mechanism is based on the modifica- 
tion of the non-interacting spectrum of the surface states 
within the bounds set by the topological bulk properties. 
The modification must be done in such a way that the 
local density of states is enhanced near the Dirac point. 
Such modifications will be shown to be accomplishable 
via changing the material properties close to the surface, 
for instance by growing a thin layer of a different TI on 
top of a given TI surface by molecular beam epitaxy. 

Relation of bulk and edge interactions. In TIs there are 
two energy scales competing with the Coulomb repulsion, 
modeled by a Hubbard interaction U in this work. In the 
insulating bulk this is the energy gap A. For A <U the 
bulk may enter a time-reversal symmetry-broken phase 
in which the original topological classification becomes 
meaningless. On the other hand, the gapless surface 
states become correlated if U Hvp/a, with vp the 
Fermi velocity of the surface Dirac cone and a the lattice 
constant (a = 1 in the following). This work focusses on 
the regime of an effectively non-interacting bulk, with a 
strongly interacting surface. Usually, however, the bulk 
gap A oc hvp/a is proportional to the surface Fermi ve- 
locity vp, because the surface states connect valence and 
conduction band. It is not obvious that a regime of a 
weakly interacting bulk and a strongly interacting sur- 
face (i.e., /S.^ U ^ hvp/a) is reachable. 

The above argument for A cx uf relies on the assump- 
tion that the linear term in the surface spectrum is dom- 
inant. This, however, is not enforced by the nontrivial 
topology. The much discussed surface Dirac cones are 
only the leading terms in the expansion of the surface 
states' energy spectrum. In general there are also higher 
order terms and the corresponding Hamiltonian of the 
surface states is of the form 
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with k = {kx, ky) the surface momentum and the spin 
Pauli matrices. Note that there may also be more com- 
plicated fc"^ terms breaking the rotational symmetry 
Those terms are not considered in this work as they do 
not affect the general idea discussed here. 
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Assuming that the surface states exist for |k| < A ^ 
1/a, the bulk gap A ~ w^A + gsA^ + O(A^). This shows 
that Vp can in principle be varied independently from the 
bulk gap. In the remainder of this work the bulk gap A 
is assumed to be fixed so that = AA~'^ — vf-A~'^ is a 
function of vf- How these cubic terms may be manipu- 
lated in actual materials will be discussed below. 

Interaction effects. The Coulomb repulsion in the sur- 
face states is modeled by a Hubbard-like interaction 



pcnds on the spin polarization m and is given by 
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where the fermionic operator cta annihilates an electron 
with spin a and momentum k in a surface state. — > oo 
is the surface area. The interaction vertex r(k, k',q) 
encodes the localization properties of the surface states 
in the direction perpendicular to the TI surface. Hi is 
inspired by the effective model for interacting graphene 
edge states [l^. The exact form of the vertex function 
r depends on the details of the TI surface, but its most 
important and generic feature is that it reduces to zero if 
one of the momenta k -I- q, k, k' — q, k' approaches the ul- 
traviolet cutoff A, at which the surface states merge into 
the bulk bands. At this momentum, the transverse wave 
function is completely delocalized into the bulk. This 
strong momentum dependence of the interaction vertex 
is a general feature of topological confinement. Following 
Ref. [llj, r is modeled by 



r(k, k', q) = AA(k + q)AA(k)A/'(k' - q)A/'(k'), (3) 



with 7V(k) = yjl - |k|/A playing the role of the 
momentum-dependent wave function weight at the sur- 
face. The larger A/'(k) the more localized is the sur- 
face state in the direction perpendicular to the surface; 
and more localization means stronger Coulomb repulsion. 
Note also that due to the fermion doubling theorem, the 
theory of one single TI surface cannot be formulated on 
a real space lattice. 

In this work the interaction Hi is treated in mean-field 
approximation. Since the SU(2) symmetry is broken by 
the SOI, it is most important to use an SU(2) invariant 
mean-field decoupling of i.e.. 
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with the components of the spin polarization m 

"V = Xl-'^^('')^^k'T'5aa'Ck^')' fi^X,y,Z. (5) 

kfjcr' 

It is most convenient to formulate this mean-field ap- 
proximation in terms of a variational principle. The total 
ground state energy of the interacting surface states de- 
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e,(k,m)e[ei.(m)-e,(k,m)] (6) 



with the single-particle energies 



e±(k,m) = ± 
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and jx,y = {vf + gi,k'^)kx,y, 7z = and /i^, = 
?7mpA/'^(k)/2. The Fermi energy ej;-(m) is fixed by re- 
quiring a constant number of electrons Nf, in the N sur- 
face states. The fraction of occupied states N^/N = 
(1 -I- /)/2 is characterized by the filling factor /. For 
half-filling, / = 0. For / = 1 (/ = 0) all surface states 
are filled (empty). The integral (|6]) cannot be solved 
exactly for general m, but the expansion of Etotiina.) at 
m = is accessible. In the following, the two opposite 
limits gs — > (normal Dirac cone) and vf (Dirac 
hourglass) are discussed. 

The critical interaction strength Uc, above which a fi- 
nite spin polarization m 7^ is favored, is calculated by 
solving d^^Etot{ma) — 0. Because of the broken spin- 
rotation symmetry [SU(2)]; Uc depends on whether m 
is in the xy plane (Uc.xy) or in z direction (Uc.z)- If 
Uc.z < Uc^xy there is a second order phase transition at 
U = Uc,z to a spin-polarized phase with Ising anisotropy 
niz ^ 0. In this phase the bulk-boundary correspondence 
breaks down: the surface Dirac cone is gapped although 
the bulk band structure predicts gapless surface states. 

For the normal Dirac cone (33 = 0), the critical inter- 
action strengths for xy- and z-polarization are 
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respectively. For half-filling (/ = 0), Uc.xy > Uc.z so that 
the magnetic state has an Ising anisotropy. For larger 
fillings the xy instability becomes dominant. Thus, there 
is a critical filling fc = 1/16 above which the correlated 
Dirac cone is easy plane. Figure [T]|^a) shows the phase 
diagram of the normal Dirac cone. The 2nd order phase 
boundaries are complemented by a 1st order transition 
line between the xy and the Ising phase. 

For the normal Dirac cone without cubic terms in the 
spectrum, the critical interactions have an explicit UV 
cutoff (A) dependence. This is a rather general feature 
of Dirac physics and is also frequently observed in the 



context of graphene (see, e.g., Refs. [12 and Il3l). The 
presence of the UV cutoff may be traced back to the 
linear energy dependence of the density of states. 

For the Dirac hourglass (.gsA^ ^ vf), the exact for- 
mulas for the Uc.z and Uc.xy ai'c accessible but rather 
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FIG. 1. (Color online) Phase diagram of the normal Dirac 
cone. The green curves are the analytical formulas for the 
critical lines [Eq. The order of the phase transitions 

are indicated. One movie in the supplemental material [13 ] 
shows the ground state energies along the cut indicated by 
the dashed grey line. 



involved [IJ]. The critical filling /c, at vifhich the XY 
phase sets in, is shown in the inset of Fig. [2j It re- 
mains finite for any vp and the corresponding critical 
interaction Uc{fc) = Uc.zifc) = Uc,xy{fc) is large for all 
Vp. Thus in the experimentally relevant limits of small 
U,vf, f the Ising instability is dominant so that we may 
focus on J7c,z- The exact formula for C/c,z for all vp is 
plotted for half- filling in Fig. [5] It vanishes for vp ^ 
and in this limit 



UcAf = 0) 



_ , (9) 
is an excellent approximation (dashed line in Fig. [5]). 
This is one of the main results of this work. For arbitrary 
filling / > the critical point approximately follows a 
scaling form 

UcAf) - UcAf = o)(i + + /*) + o{rf'"- (10) 

with the renormalized filling /* = {A-Kvpkp /Uc,z{f ~ 
0))^. As shown in Fig. [31 Eq. ([TUj) is a good approxi- 
mation to the true phase boundary. It should be noted 
that, unlike in the pure Dirac case ((73 = 0), A — >■ 00 is in 
principle a well-behaved limit for the Dirac hourglass; the 
terms regularize the UV divergence in the integrals. 
Nevertheless, the logarithmic corrections in Eq. ^ arc 
still significant down to vp/gsA'^ ~ 10~^. 

Properties of the magnetic state. The expansion of 
i5tot(ni) up to 4th order in m gives information about 
the mean-field critical exponent /3 of the transitions. 
For half- filling (/ = 0) the Ising transition has /? = 1. 
This anomalous exponent is due to the linear energy- 
dependence of the density of states near the charge neu- 
trality point. For / 7^ 0, the Ising and XY transitions 
have /? = 1/2, since in this case the density of states is 
finite at the Fermi level. 

I now turn to the question if the above mean-field state- 
ments are valid. The effective theory for the surface spin- 
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FIG. 2. Suppression of the critical interaction and stabiliza- 
tion of the Ising phase by lowering vp: Uc,z is shown for 
half-filling (/ = 0). The curve labeled by Uc{fc) shows the 
interaction strength at which the transition to the XY phase 
occurs. The inset shows the critical filling fc above which 
the spin polarization is in-plane. The dashed line shows the 
approximation Eq. ([9]). A = 1 and A = 1 in this plot. 
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FIG. 3. (Color online) Phase boundary between the param- 
agnet and the Ising ferromagnet phase for different vp = 2^" 
with V from 1 (red) to 12 (blue), thus varying over three or- 
ders of magnitude. The thick black line is the approximate 
scaling form Eq. (jlOp . The inset shows how the real filling / 
decreases for df — ^ 0. A = 1 and A = 1 in this plot. 
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, Ckcr' in the XY phase is 
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DS^^ with D a positive constant (see Ref. 



151), 



and (S^) > for f7 > Uc,xy The mean- field approach 
in the present work corresponds to treating S as a classi- 
cal vector, which may point in any in-plane direction in 
the XY phase. A simple quantum-mechanical treatment 
of -ff^f^, however, gives rise to a non-degenerate ground 
state and zero spin-polarization (5"^) = (// = x,y,z), 



S ) 7^ [15|. In the thermodynamic limit 



but with 

(S^) — >■ 00. Thus, the XY phase describes a large super- 
spin pointing in no direction, which is in a sense the least 
classical scenario. This is an indication that the mean- 
field theory of the XY phase does not describe all mag- 
netic properties correctly. Nevertheless, small islands at 
the TI surface, for which mean-field theory predicts an 
in-plane spin polarization, are expected to show mag- 
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FIG. 4. (Color online) Non-interacting surface state spec- 
tra in homogeneous slabs of 3D TIs for different A2 and A4 
parameters [(a) and (b)]. The Fermi velocity of the surface 
states has opposite signs in (a) and (b). Part (c) shows the 
surface state spectrum for a layer structure in z direction with 
a homogeneous x and y direction. The other parameters are 
equal in all plots: M = -0.5, B = -0.25, Ai = 3, A3 = 0. 



netic characteristics similar to the superspins at the ends 
of zigzag carbon nanotubes [l5l | . 

In the Ising phase the effective theory is H^^f = D' 
with D' <Q and the ground state is two-fold degenerate. 
In this case, the thermodynamic limit corresponds to the 
classical limit of a large superspin which points in ±z 
direction. Furthermore, it is well known that a 2D Ising 
fcrromagnet may develop long range order at finite tem- 
peratures. Therefore it can be expected that the Ising 
phase is thcrmodynamically stable. 

How to generate the cubic terms. In the standard 
k-space model Hamiltonian for a 3D TI [l^ there 
are two relevant types of terms. The term of the form 
Hq = TzAA{}<i), with the Pauli matrices acting in or- 
bital space and Al(k) = M — 2B{2> — Y^f^cosk^j), is an 
identity in spin space and is essentially responsible for 



creating a gap in the band structure. T'hc term of the 
form Hso ~ Tri-s^^p(kj is, among other things, rcsponsi- 
blc for the spectrum of the surface states Following the 



(11) 



where is a four-component spinor @, UBIj n runs over 
a cubic lattice and 8 connects a site to its neighbors. For 
nearest-neighbor SO hopping only, Eq. ([TT]) reduces to 
the usual ^^(k) = Aismk^. However, if 1st, 2nd and 
4th neighbor SO hopping is taken into account [13] , then 
A^(k) =:sinA:^(Ai + 2A2X;^^^cos(fc^)) + ^4sin2A:^. For 
Ai + 2A2 + 2A4 = and Ai - 4yl2 + 8^4 = one finds a 
cubic surface spectrum A^{k) cx k^]s?, with k = (fc^, ky). 

In real materials it is difficult to design the hopping 
amplitudes, which makes homogeneous TIs with vp — Q 
unfeasible. A more realistic approach for tuning vp to 
zero is based on attaching a thin layer of a TI with oppo- 
site sign of vp to the surface of a given TI. If the thickness 
of this layer is chosen properly (layer thicknesses are well 
controlable with molecular beam epitaxy techniques) , the 
opposite linear terms average to zero and only the cubic 
terms are left. This is illustrated in Fig. 01 where the 
numerically calculated non-interacting surface spectra of 
homogeneous TIs with different material parameters [(a) 
and (b)] arc compared to a layer structure [part (c)]. 

Conclusion. A mechanism for tuning surface states of a 
3D TI to a strongly correlated regime has been discussed. 
Cubic terms cx are introduced in the surface state 
spectrum in such a way that the Fermi velocity vp may be 
tuned to zero while keeping a large bulk energy gap. This 
limit is characterized by strongly correlated surface states 
with an effectively non-interacting and topologically non- 
trivial bulk. The strong surface correlations open a gap 
in the surface states, even though the topological bulk 
classification wrongly predicts gapless surface states. A 
mechanism for generating and tuning the cubic terms in 
an actual TI has been discussed. 

I am grateful for stimulating discussions with C. Hon- 
erkamp, T. C. Lang, and S. Wessel. 
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